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Abstract 

o 

CN| ' We consider optimal scalar quantization with rth power distortion and constrained Renyi entropy 

of order a. For sources with absolutely continuous distributions the high rate asymptotics of the 
quantizer distortion has long been known for a = (fixed-rate quantization) and a = 1 (entropy- 
\ constrained quantization). These results have recently been extended to quantization with Renyi en- 

tropy constraint of order a > r+1. Here we consider the more challenging case a £ [-co, 0) U (0, 1) 
and for a large class of absolutely continuous source distributions we determine the sharp asymptotics 
of the optimal quantization distortion. The achievability proof is based on finding (asymptotically) 
Q , optimal quantizers via the companding approach, and is thus constructive. 

Index Terms: companding, high-resolution asymptotics, optimal quantization, Renyi entropy 

^ ' 1 Introduction 

00 

With the exception of a few very special source distributions the exact analysis of the performance of 
optimal quantizers is a notoriously hard problem. The asymptotic theory of quantization facilitates such 
analyses by assuming that the quantizer operates at asymptotically high rates. The seminal work by 
Zador [31] determined the asymptotic behavior of the minimum quantizer distortion under a consttaint 



on either the log-cardinality of the quantizer codebook (fixed-rate quantization) or the Shannon entropy 
of the quantizer output (entropy-constrained quantization). (See the article by Gray and Neuhoff |[T2ll 
for a historical overview and related results.) Zador's results were later clarified and generalized by 
Bucklew and Wise [5] and Graf and Luschgy [9] for the fixed-rate case, and by Gray et al. ifTTTl for the 
entropy-constrained case. 

Recently, approaches that incorporate both the fixed and entropy-constrained cases have been sug- 
gested. In [10] a Lagrangian formulation is developed which puts a simultaneous constraints on entropy 
and codebook size, including fixed-rate and entropy-constrained quantization as special cases. Another 



W. Kreitmeier is with the Department of Informatics and Mathematics, University of Passau, InnstraBe 33, 94032 Passau, 
Germany (email: wolfgang.kreitmeier@uni-passau.de). T. Linder is with the Department of Mathematics and 
Statistics, Queen's University, Kingston, Ontario, Canada K7L 3N6 (email: lindergmast . queensu . ca). 

This research was supported in part by the German Research Foundation (DFG) and the Natural Sciences and Engineering 
Research Council (NSERC) of Canada. 



1 



approach that has been suggested in [10] and further developed in |[T8l [T9ll uses the Renyi entropy of 
order a of the quantizer output as (generalized) rate. One obtains fixed-rate quantization for a = 0, 
while a = 1 yields the usual (Shannon) entropy-constrained quantization approach. 

The choice of Renyi entropy as the quantizer's rate can be motivated from a purely mathematical 
viewpoint. In the axiomatic approach to defining entropy, Renyi 's entropy is a canonical extension of 
Shannon-entropy, satisfying fewer of the entropy axioms |[26l [T|. From a more practical point of view, the 
use of Renyi entropy as quantizer rate is supported by Campbell's work |6j, who considered variable- 
length lossless codes with exponentially weighted average codeword length and showed that Renyi's 
entropy plays an analogous role to Shannon entropy in this more general setting. Further results on 
lossless coding for Renyi entropy were obtained in (24J. Jelinek |[T6ll showed that Renyi's entropy (of an 
appropriate order a G (0, 1)) of a variable-length lossless code determines the encoding rate for a given 
reliability (exponential decrease of probability) of buffer overflow when the codewords are transmitted 
over a noiseless channel at a fixed per symbol rate. At least in such situations, measuring the quantizer's 
rate by Renyi's entropy is operationally justified. An overview of related results can be found in [2]. The 
diverse uses of Renyi's entropy (and differential entropy) in emerging fields such as quantum information 
theory (e.g. |[T5l ). statistical learning (e.g. ifTTj ). bioinformatics (e.g. 112111 ). etc., may also provide future 
motivation for this rate concept. 

The only available general result on quantization with Renyi entropy constraint appears to be |[T8l 
where the sharp asymptotic behavior of the rth power distortion of optimal d-dimensional vector quan- 
tizers has been derived for a G [1 + r/d, oo]. The proof shows that for these a values the optimal 
quantization error is asymptotically determined by the distortion of a ball with appropriate radius around 
the most likely values of the source distribution. Thus it suffices to evaluate the rth moment of this ball 
(see lfT8l Theorem 4.3]), which remarkably simplifies the derivation and makes the case a > 1 + r/d 
quite unique. In the classical (a = and a = 1) settings, the contributions of the codecells of an optimal 
quantizer to the overall distortion are asymptotically of the same order. Bounds on the optimal perfor- 
mance in [ 1 8 ] suggest a similar situation for a < 1 + r/d, making the problem more challenging than 
the case a > 1 + r/d. 

In this paper, at the price of restricting the treatment to the scalar (d = 1) case, we are able to 
determine the asymptotics of the optimal quantization error under a Renyi entropy constraint of order a £ 
[— oo, 0) U (0, 1) for a fairly large class of source densities. The achievability part of the proof (providing 
a sharp upper bound on the asymptotic performance) is constructive via companding quantization. In 
particular, we determine the optimal point density function for each a G [— oo, 1 + r) and provide 
rigorous performance guarantees for the associated companding quantizers (for a = and a = 1, these 
results have of course been known). Matching lower bounds are provided for a E [— oo, 0)U(0, 1), which 
leaves only the case a G (1, 1 + r) open. We note that in proving the matching lower bounds, one cannot 
simply apply the techniques established for a = or a = 1. In our case the distortion and Renyi entropy 
of a quantizer must be simultaneously controlled, a difficulty not encountered in fixed-rate quantization. 
Similarly, the Lagrangian formulation that facilitated the corrected proof of Zador's entropy-constrained 
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quantization result in ifTTI cannot be used since it relies on the special functional form of the Shannon 
entropy. On the other hand, using the monotonicity in a of the optimal quantization error, one can show 
that our results imply the well-known asymptotics for a G {0, 1}, at least for the special class of scalar 
distributions we consider. 

The paper is organized as follows. In Section [2] we introduce the quantization problem under a 
Renyi entropy constraint and review some definitions and notation. In Section [3j after summarizing 
some related work, we state our main result. The next three sections are devoted to developing the 
machinery needed in the proof. Section|4]presents results on the asymptotic distortion and Renyi entropy 
of companding quantizers, which, with the proper choice of the compressor function in a Bennett-like 
integral, will turn out to be (asymptotically) optimal. In Section [5] technical results needed mostly for 
establishing lower bounds are developed. Section [6] presents upper and lower bounds on the optimal 
quantization error for mixture distributions. Section |7] contains the proof of the main results. Section [8] 
contains concluding remarks and a discussion on extending the results to vector quantization. All the 
longer, technical proofs of the auxiliary results are relegated to the appendices. 



2 Preliminaries and notation 



We begin with the definition of Renyi entropy of order a. 



Definition 2.1. Let N := {1, 2, . . .}. Let a G [— oo, oo] and p = (pi,P2, • • •) £ [0, 1] N be a probability 
vector, i.e., Y^LiPi = 1- Th e Renyi entropy of order a, H a (p) G [0, oo], is defined as (see / |26|/ , /[7] 
Definition 5.2.35] and Kl4\ p. 1]) 



H a (p) 



a G (— oo, oo) \ {1} 
a = 1 



\i:pi>0 J 

oo 

- E Pi l, =>gpi, 
i=i 

- log (maxjj?j : i G N}) , a = oo 

- log (inffe : i G N,pj > 0}) , a = -oo. 



We use the conventions • log := and 0° := 0. All logarithms are to the base e. 
Remark 2.2. (a) With these conventions we obtain 

H°(p) = log (card{i G N : p, L > 0}) , 

where card denotes cardinality. Using l'Hospital's rule it is easy to see, that the case a = 1 follows from 
the case a ^ 1 by taking the limit a — > 1. (see, e.g., [1 , Remark 5.2.34]). Moreover, one has 



lim H a (p) = H°°{p) and lim H a (p) = H-°°{p). 



a— ¥— oo 



(1) 
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(b) We note that the usual definition of Renyi entropy is restricted to nonnegative values of the order 
a. However, it will turn out that the case a < can be handled without too much additional technical 
difficulties, and we believe that this generalization may turn out to have useful implications. 

Now let d £ N and X be an Revalued random variable with distribution p. Let I C N and S = 
{Si : i € 1} be a countable and Borel measurable partition of M. d . Moreover let C = {cj : i € 1} be a 
countable set of distinct points in R d . Then (S, C) defines a quantizer q : R d — > C such that 

q(x) = Ci if and only if x € S{. 

We call C the codebook and the Cj the codepoints. Each Si € S is called codecell. Clearly, C = q(R d ) 
(the range of q). Moreover, 

S = {q- 1 (z):zeq(R d )} 

where q~ x {z) = {x € R d : q(x) = z}. Let denote the set of all quantizers on R d , i.e., the set of all 
Borel-measurable mappings q : M. d — > R d with a countable number of codepoints The discrete 

random variable q(X) is a quantized version of the random variable X whose distribution is denoted by 
H o q^ 1 . In measure-theoretical terms the image measure \i o q^ 1 has a countable support and defines 
an approximation of \i, the so-called quantization of fj, by q. With any enumeration {ii, «2, • • •} of I we 
define 

H%(q) = H<*(fi(S h ),KSi 2 ),...) (2) 

as the Renyi entropy of order a of q with respect to fj,. We intend to quantify the error in approximating 
the original distribution \i with its quantized version \i o q^ 1 . To this end let || • || be any norm on M. d 
and p : [0, oo) — > [0, oo) a strictly increasing function. For q G we measure the approximation error 
between X and q(X), resp. p and p o q^ 1 , also called the quantizer distortion, as 

D^q)=Ep(\\X-q(X)\\)= J p{\\x - q(x)\\) dp(x). 

For any R > we define 

D«(R) = inf {D^q) : q G Q d , H°(q) < R}, (3) 

the optimal quantization distortion of p under Renyi a-entropy bound R. We note that D®(R) is a 
nonincreasing function of a (see Lemma |23T ). 

We call a quantizer q optimal for p under the entropy constraint R if D^(q) = D®(R) and H®(q) < 
R. In the rest of this paper we focus on the one-dimensional case (scalar quantizers, d = 1) and the 
so-called rth power distortion measure p(x) = x r , where r > 1. Thus the distortion of quantizer q £ Qi 
is given by 

D fl (q) = E\X-q(X)\ r = [ \x - q(x)\ r dp{x). 
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For simplicity we write Q\ = Q. Also, let Q c C Q denote the set of all scalar quantizers with 
finitely many codecells, each of which is an interval, and such that every codepoint lies in the closure of 



the corresponding codecell. The following lemma (proved in Appendix A ) presents two key properties 
of optimal quantization under Renyi entropy constraint. 

Lemma 2.3. For all R>0 and a, (3 G [—00, 00] with f3<a,we have 

D?(R)>D*(R). (4) 
Assume that E\X\ r < 00 and p is nonatomic. Then for all R> and a G [—00, 0], we have 

D«(R) = inf {£>„(?) : q G Q c , H%{q) < R} (5) 

while for all a G (0, 00], 

DftR) = hxl{D^q) :qeQ c , H«{q) = R}. (6) 

The second statement of the lemma says that under the given conditions the optimum quantizer 
performance can be approached arbitrarily closely by quantizers in Q c . For this reason, in the rest of 
the paper all quantizers will be assumed to belong to Q c ; in particular, we only consider quantizers 
with finitely many interval cells. According to ©, when a G (0, 00] it suffices to consider only those 
quantizers in Q c whose entropy attains R. 

From [18, Thm. 5.2] it is known that for a G [0, 1] the product e rR D^(R) remains bounded and is 
bounded away from zero as R — > 00. This motivates the following notion of quantizer optimality that 
will play an important role in our work. 

Definition 2.4. Let {q n ) n ^n C Q be a sequence of quantizers such that H®(q n ) — > 00 as n — > 00. If 

e rR D®(R) — > cas R — » 00 for some c G (0, 00) and 

lim e rH ^D^(q n ) = c, (7) 

n— >oo 

then we call (q , n )neN an asymptotically optimal sequence of quantizers for ji. 

We denote by A the one-dimensional Lebesgue measure. For a measurable real function /oni and 
measurable nonempty set A C R, essinf^/ = sup{6 : \{{x G A : f(x) < b}) = 0} denotes that 
the essential infimum of / on A. Similarly, esssup^ / = inf{6 : \({x G A : f(x) > b}) = 0} is the 
essential supremum of / on A. We let supp(^) denote the support of p, defined by 

supp(^) = {x : p{(x — e, x + e)) > for all e > 0}. 

Note that supp(/u) is the smallest closed set whose complement has p measure zero. We will often deal 
with the situation where supp(/i) is contained in a bounded interval /. In such cases, we usually leave a 
quantizer q G Q undefined outside /, as we may since p(R\I) = 0. 
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Let Z denote the set of all integers and assume A > 0. The infinite-level uniform quantizer q& on 
R has codecells {(iA, [i + 1)A] : i € Z} and corresponding codepoints that are the midpoints of the 
associated cells, so that <7a0*0 = (i + 1/2)A if and only if x G (iA, (i + 1)A]. 



3 Main results 

First we summarize the known results regarding the sharp high-rate asymptotics of the distortion of 
optimal scalar quantizers. In order to unify the treatment, we reformulate the classical (resolution and 
entropy) rate constraints in terms of the Renyi entropy with appropriate order. For r > we let 

C(r) ' 



(1 + r)2 r ' 

Theorem 3.1 ( OTl [5] [9j [TU (HI). Let r > 1 and \i = fj, a + p, s be the Lebesgue decomposition of 
distribution n of the scalar random variable X with respect to the one-dimensional Lebesgue measure 
A, where fj, a denotes the absolutely continuous part and fi s the singular part of/j,. Assume that fi a (M) > 
and let f = be the density of /i a . 

(i) If a = and E\X\ r+s < oo for some 5 > 0, then 

lim o e rR D^R) = C(r) (| dX}^ . (8) 

(ii) If a = 1, // S (M) = 0, f f log / d\ exists and is finite, and H^q^) < oo for some A > 0, then 

lim e rR Dl(R) = C{r)e~ r f fl ° si ~ dX . (9) 

(Hi) If a € [1 + r, oo], p, s (M) = 0, E\X\ r+s < oo for some 5 > 0, and ess sup K f < oo, then 

lim e {1+r) ^ a)R D a JR) = C{r) (esssup K /)- r , 

where /3(a) = (a — l)/a if a € [1 + r, oo) and /3(a) = 1 if a = oo. 

Note that / is a probability density function if and only if /U S (R) = 0. Part (i) of the theorem 
is originally due to Zador [31] who considered the multidimensional case; corrected and generalized 
proofs were given by Bucklew and Wise [5 ] and Graf and Luschgy [9]. Part (ii) is also due to Zador OTTl 
with corrections and generalizations by Gray et al. (TT). Part (iii) is due to Kreitmeier [ 18 ] who also gave 
upper and lower bounds for the case a E (1, 1 + r). 

Definition 3.2. A one-dimensional probability density function f is called weakly unimodal if f is con- 
tinuous on its support and there exists an Iq > such that {x : f(x) > 1} is a compact interval for every 
I G (0,Zo). 
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Remark 3.3. Note if / is weakly unimodal density, then it is bounded and its support is a (possibly 
unbounded) interval. Clearly, all continuous unimodal densities are weakly unimodal. Thus the class of 
weakly unimodal densities includes most parametric source density classes commonly used in modeling 
information sources such as exponential, Laplacian, Gaussian, and generalized Gaussian densities. 

For a € (—00, r + 1) \ {1} we define 

1 - a + ar 1 — a + r 

a% = ■ , a 2 = — . (10) 

1 — a + r 1 — a 

The following is the main result of the paper. 

Theorem 3.4. Let r > 1 and assume that the distribution n of X is absolutely continuous with respect 
to X having density f. Assume that esssup K / < 00 and let M = (inf(supp(/i))), sup(supp(/i))). In 
either of the following cases: 

(i) a € (0, 1), E\X\ r+5 < 00 for some 5 > 0, and f is weakly unimodal, 

(ii) a € (—00, 0), ess inf m f > and f is continuous on M, 
we have 



£»2 

l e rR D a ,(R) = C(r) ( / f ai dX 1 
If ess infjvf / > and f is continuous on M, then 



lim e rR D«(R) = C(r) ( [ d\) ' . (11) 

R^oo f \J M J 



lim e rR D^(R) = C(r) ( [ f^dx) . (12) 

The proof of the theorem is given in Section|7] Upper bounds will be established using a companding 
approach, while matching lower bounds are developed by considering increasingly more general classes 
of source densities. 

Remark 3.5. (a) Note that if we formally substitute a = in (fTTb . it reduces to ([8]). Moreover, it is easy 
to show that (TTTT t reduces to Q if a — > 1. Due to monotonicity of the quantization error (Lemma l231 ) and 
by the upper bound for the quantization error for a € [—00, 1 + r) (Corollary 14. 11 1 ) one can rigorously 
show that the known asymptotics for a € {0, 1} also follow from Theorem |3.41 at least in the scalar case 
and under our restrictions on the source density. 

(b) The results of the theorem can be expressed in terms of the Renyi differential entropy h a ([i) = 
log(J f a dX) of order a ^ 1. It is easy to check that (fTTT > can be rewritten as 

lim e rR D a JR) = C{r)e rhai ^. (13) 

Setting ai = linio^^oo ^Sfa+r = 1 — ^ for a = —00, we also obtain (Tf2l from the above expression. 
Also, for a = we have ax = j^-p, and (Tf3l) reduces to ([8]); while for a = 1, we have ax = 1, and 
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we formally get back © since lim ai _>.i h ai (/j.) = /i 1 ( / u) = — / /log/dA (cf. Section |4~2~1). Thus (fT3l) 
expresses the old and the new asymptotic results in a unified form. 

(c) Since essinf^ / > the right hand side of (fl2l) is finite. For the same reason, the right hand side 
of (fTTT) is finite for all a < 0. For a G [0, 1) the right hand side of (TTTT t can be shown to be finite by an 
application of Holder's inequality as in (9j Remark 6.3 (a)]. 

(d) The weak unimodality and continuity conditions on / are the results of our approximation techniques 
in proving lower bounds and are probably not necessary. In fact, with a a little tweaking of the compand- 
ing approach in the next section one can show that the right hand sides of (fTTT > and (fT2l) still upper bound 
the asymptotic performance if these conditions are dropped. 

(e) Note that condition (ii) implies (i). Also, the right hand side of (TTTb converges to the right hand side 
of (fT2l) as a — s> — oo. 

(f) The condition r > 1 is needed in the proof of the lower bounds on D®(R) where lfl9l Thm. 3.1] is 
invoked (see Proposition 17.21 ). The upper bounds only need r > 1 (see Section |4]). 



4 Distortion and Renyi entropy asymptotics of companding quantizers 
4.1 Companding quantizers 

Let N > 2 and Qn G Q denote the iV-level uniform scalar quantizer with step size 1/N for sources 
supported in the unit interval [0, 1] defined by Qn{x) = 1/2N if x G [0, 1/N] and 



„ , s i — 1 , 1 . r (i — li 

Qn{X) = —r— + — if X G 



i = 2,...,JV. (14) 



iV 2N \ N N 

The compressor G derived from a probability density g on the real line is the function 

G(x) = [ X g(y)d\{y). (15) 

J — oo 

Thus the increasing function G : M — > [0, 1] is the cumulative distribution function associated with the 
density g. The generalized inverse G of G is defined by 

:= sup{x : G{x) < y} = max{x : G(x) < y} 

for y G (0, 1). Note that if g is positive almost everywhere with respect to A (a.e. for short), then G is 
strictly increasing and G is its (ordinary) inverse. 

In this paper we will work only with compressor densities g having compact support, i.e., if v denotes 
the measure induced by g, then supp(z^) is bounded. Thus we can extend the definition of G onto [0, 1] 
by letting 

G(0) := min{supp(i^)} > — oo and G(l) := max{supp(i^)} < oo. 
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The iV-level companding quantizer Q 9j n associated with g is defined on [G(0), G(l)] by 



Q g>N (x) = G(Q N (G(x))). 
Note that the codecells of Q 9i n are N intervals Ii,n, ■ ■ ■ , In,n with = [G(0), G(l/N)] and 

h,N = (G((i - 1)/N), G(i/N)], i = 2, . . . , N. 
The corresponding quantization points are G((2i — l)/2N),i = 1, . . . ,N. 

Remark 4.1. (a) The function g is often called the point density for Q 9 ,n{x) since it has the property 
that for any a < b, 

1 f b 

lim — card(Q 9iA r((a,6))) = / g(x)dX{x). 

N^oo 1\ J a 

(b) If P/v is an arbitrary AMevel quantizer on R having convex (interval) codecells, then it can be im- 
plemented as a companding quantizer. In particular, there exists a positive point density g such that 
Pn(x) = Q g ,N(x) for all (except perhaps a finite number ofjiGf (any x such that Pn{x) / Q g ,N{x) 
is a cell boundary for both quantizers). 

The following result represents the error asymptotics of the compander if the number of output levels 
increases without bound. The result originates with Bennett [3] for r = 2 and has appeared in the 
literature in several different forms (but most often without precise conditions and a rigorous proof); see 
irT2l for a historical overview. The proof is given in Appendix A and follows the development in lF22l 



which gives a rigorous proof for the limit (fTBT ) under different conditions that include the continuity of g 
and certain tail conditions, but allow / and g to have unbounded support. 

Proposition 4.2. Let X be a random variable with distribution [i which is absolutely continuous with 
respect to A and let f denote its density. Let G be a compressor with point density g. Assume that the 
support of p, is included in a compact interval I such that essinf/gf > and g(x) = a.e. on R \ J. 
Then for r > 1, 

lim N r D„(Q g , N ) = C{r) [ L dX. (16) 

J j g r 

Remark 4.3. Since essinf/g > we know that \i is absolutely continuous with respect to g X and 
[ T dX < 00. 

Jig 

4.2 Renyi entropy asymptotics of companding quantizers 

In order to be able to construct asymptotically optimal companding quantizers (cf. (|7])), in addition to 
the asymptotic distortion, we also have to control the quantizer's entropy, at least for high rates. In this 
section we derive a result (Proposition 14.81) which asymptotically describes the Renyi entropy of the 
compander as a function of the number of quantization points. Let 1a denote the indicator function of 

A c R. 
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Definition 4.4. Let p, be absolutely continuous with respect to A with density f and define M 
(inf(supp(//)), sup(supp(/x))). Let a G [— 00,00] and assume that 

(i) lsu PP (^)/ a is integrable if a G (-00, 00) \ {1}, 

(ii) essinfjvf / > if a = —00, 

( Hi) f log / is integrable if a = 1, 
(iv) esssup R / < 00 if a = +00. 
Then the Renyi differential entropy of order a of p is defined by 

'l^M^pp^rdA), «g(-oo,oo)\{1} 

-J/log/tiA, a = 1 

-log(esssup R /), a = 00 

— log(essinfj\/ /), a = — 00. 



Remark 4.5. Just as in the case of Renyi entropy (see Remark I2l2l ) the mapping [—00, 00] 3 a — > h a (/i) 
is continuous for the differential entropy. 

Recall that q& denotes the infinite-level uniform quantizer with step-size A > 0. Recall M from 
Definition @3 and let A(A, M) = {a G q&(R) : q~^{a) C M} and 

aeA(A,M) 

The following result is due to Renyi [27 ] and Csiszar JH for a € (0, 00). The proof for a G [—00, 0] 



is given in Appendix A 



Lemma 4.6. Let \i be absolutely continuous with respect to A having density f. Let a G [—00, 00) and 
assume that the Renyi differential entropy of order a of p, exists and is finite. Assume that H^(q^) < 00 
for some A > 0. If a G (—00, 00), then 

lim (fl£(g A ) + log(A)) =h a (p). 

Moreover, 

lim (iJ--°°(g A ,M) + log(A)) = h-°°(jM). 

Next we define the Renyi relative entropy between two probability measures for the case where both 
have densities. 

Definition 4.7. Let \x and v be probability measures which are absolutely continuous with respect to 
A. Denote by f and g the densities of p and v. Moreover, assume that fi is absolutely continuous with 
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respect to v and, therefore, we assume w.l.o.g. that {g = 0} C {/ = 0}. Setting 



E = {/ > 0} and M = (inf (supp(/i)), sup (supp (//))) 
the Renyi relative entropy of order a between the distributions p and v is defined as 

^log^/V^^Y «G(-oo,oo)\{l} 



V a (p\\v) 



J B /log|dA, a=l (17) 

log(esssup£ i), a = oo 

log(essinfM £), a = — oo. 

(For D_ 00 (//||z/) to be well defined, we need the condition ess inf m f > O.J 

The following result determines the asymptotics of the Renyi entropy of a companding quantizer. 
Proposition 4.8. Let a G [—00,00). Suppose p and v are as in Definition 14. 7\ and T> n (p \ \ u) < 00. Then 



lim fe(Q S)JV )-logiV) =-P«0*||i/ 



)■ 

Remark 4.9. (a) For the sake of distortion analysis we previously specified that g has bounded support, 
but in this proposition the only condition on / and g is the finiteness of V a {p\\v). 
(b) In a sense, the proposition generalizes Lemma 1431 Indeed, if the support of p is included in a 
compact interval / and g is the uniform density on I, then Q 9i n is the uniform quantizer of step-size 
A at = \{T)/N over I, and the proposition reduces to Lemma |4~6l (for the sequence of step-sizes An). 

Proof. Recall the definition of the compressor G from ( fTBT ). We proceed in two steps. 

1. We show that h a ([i o G^ 1 ) = —V a {p\\v) for every a G [—00, 00). 

Let a G (—00, 00) \ {1} and let fa be the density of p o G^ 1 (see Lemma lATTI in Appendix Ai. 
Definition \4A\ and Lemma lATTI imply 



h a {poG- 1 ) = — L_ log /(/ G ) a dA 
1 - a 7 

log f(f(G(y))G'(y)rdX(y) 



1 


1 




a 




1 




1 




a 




1 




1 




a 




1 




1 




a 



log / f{G{y)rg{G{y)) 1 - a G'{y)d\{y) 
log (7 /(x) a ff (x) 1 - a C /A(x) > ) = -D a ( M ||i/) 



where in the penultimate equality we used again the chain rule for the Lebesgue integral (see [30, Corol- 
lary 4]), which is applicable due to the monotonicity of G and the integrability of f a g l ~ a (which follows 
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from the finiteness of T> a (fi\\u)). Note that the above chain of equalities implies that (fc) a is integrable. 
One can deduce the assertion of step 1 for a € {— oo, 1} in a very similar manner. 

2. Now we prove the assertion of the proposition. Since G is increasing and continuous, G is strictly 
increasing on (0, 1). Recall the definition of Q n in (TT41) and note that Qtv = <7i/at on (0, 1). Then 

H«{Q g>N ) = H% oG -i(Q N ) (18) 

for all a G [— oo, oo). Since /i o G _1 ((0, 1)) = 1, we obtain H° oG _ 1 (Qjq) = ff a oG -i (Qi/n)- In view 
of ( fT8l ) we deduce H^(Q g ^) = H® oG -i{4i/n)- From step 1 and by the assumption we know that 
h a (n o G~ l ) is finite. Since qi/ N has no more than N cells with nonzero fi o G~ x — measure, the entropy 
^poG" 1 (.Qi/n) i s a l so always finite. Lemma 1431 and step 1 imply 



lim (H«{Q g . N ) - log JV) = lim U a G ^ (q 1/N ) - log N 

= / l Q (/ioG- 1 ) = -P a (^||z.). 



□ 



Remark 4.10. Although we do not need this fact in the sequel it is worth noting that Lemma 1461 and 
Proposition 14. 8 1 are also valid for a = oo. For example, by an application of Lebesgue's density theorem 
one can show that 

sup{/i(^ 1 (a)) : a G £a(R)} , 
hm ^ = ess sup R /, 

which yields the assertion of Lemma l4.6l for a = oo. Generalizing the proof of Proposition l4.8l to a = oo 
is straightforward. 

4.3 Optimal point densities 

Combining the previous results we can find a companding quantizer which provides an (asymptotic) 
upper bound for the optimal quantization error. Later on we will show that this quantizer is an asymptot- 
ically optimal one. Recall definition (fTOb of a± and a?,- 

Corollary 4.11. Let r > 1 and a G [— oo, 1 + r). Assume that [i is supported on a compact interval I 
and has density f such that ess inf/ / > 0. Moreover, assume that f ai is integrable if a G (1, 1 + r ) and 
f log / is integrable if a = 1. Let 



r 



(Jj f 1 ^ 2 dX)- 1 ! 1 '^, a G (-oo, 1 + r) \ {1} 

(A^))- 1 !/, a = l (19) 

/, a = -oo. 
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Then, 

' .TdA)* a€(-oo,l + r)\{l} 



lim e rH ^r,n) D [Q f N ) 
N—>oo 



C ( r y-r ff log fd\^ a = 1 (20) 



-oo. 



Proof. It is not hard to show using Holder's inequality that f ai 1/ is integrable for every a S (—00, 1 + 
r) \ {1} (cf. El Remark 6.3 (a)]). Moreover / 1-r is integrable. Clearly, / 1//a2 l/ is integrable for every 
a € (-00, 1 + r). These facts imply that /* is well defined (note that essinf/ /* > 0), j //(/*) r dX < 
00, and the integrals on the right hand side of (l20l are finite. It is also easy to check that V a (fi\\f* X) is 
finite. Thus we can apply Propositions 14. 8 1 and 14.21 We obtain 

lim e rH ^ Q r^DJQ f *, N ) = lim e~ rV ^ f '^ N r D^Q f ^ N ) 

N— s-oo N^oo 

v ' J i (f*y 

Now (fT9l) and (fTTT ) yield the assertion. □ 



Remark 4.12. For a€[— 00, 1 + r) the point density g = f* in the corollary minimizes the asymptotic 
performance lim e rH ^^s,N) ' Dn(Q g n)- F° r a = and a = 1 this optimal choice of 5 has long been 
known. In the case a G (—00, 1 + r) \ {1}, by Propositions 14.21 and 14. 8 1 the above limit is proportional to 

T 

rg X - a dx\' a [ fg- r dX 
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and Holder's inequality (for a < 1) or the reverse Holder inequality (for a G (1,1 + r)) can be used to 
show that this functional is minimized by g = /*. The resulting minimum is (J, f ai dX) a2 . The case 
a = — 00 follows by letting a — > — 00. 

5 Some important properties of optimal scalar quantization 

Define 

»(/) = ess inf supp(M) /, a(/) = ess sup supp(A() /. 

For the case a = the following result is originally due to Pierce ([25 ], [9, Lemma 6.6]). In our proof, 
given in I Appendix B[ we use a refined version provided by Luschgy and Pages ll23l Lemma 1]. 



Proposition 5.1. (i) If R > 1 and f \x\ r+/3 dfx{x) < 00 for some j3 > 0, then there exists a constant 
Co > ( which depends only on r and j3 ) such that 

f r \ r/(r+P) 

e rR D«(R) < Co ( / \x\ r+ Pdfi(x) 
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for every a G [0, oo]. 

(ii) Suppose supp(ju) is a compact interval and /x absolutely continuous with respect to A with density f. 
Assume that i(f) > 0. Then for all a < 

As an immediate consequence we obtain the following. 

Corollary 5.2. Under either condition (i) or (ii) of Proposition \5J] we have lim/j-^oo D^(R) = 0. 

Let diam(A) = sup{|x — y\ : x, y G A} denote the diameter of an arbitrary non-empty set Aci. 
The next result shows that the measure of the codecells of optimal quantizers tends to zero for absolutely 
continuous distributions. The proof, given in AppendixTJ] adopts some techniques of Gray et al. ifTTl 
Proof of Lemma 11]. 

Lemma 5.3. Let \i be absolutely continuous with respect to A having density f. Assume further either 
of the following conditions 

(i) a G [0, oo] and f \x\ r+ ^ dp(x) < oo for some /3 > 0, 

(ii) a < and supp(/i) is a compact interval and < i(f) < s(f) < oo. 

Then for every e > there exists an Rq > with the property that for every R > Rq there is a 5 > 
such that 

max{^(g _1 (a)) : a G q(R)} < e (22) 

for every q £ Q with H°(q) < R and \D fl (q) — D°(R)\ < 5. If additionally, in case (i) the support of 
p consists ofrn > 1 compact intervals I\, . . . ,I m and i(f) > 0, then in both cases (i) and (ii) we have 

max{diam(g _1 (a) fl Ij) : a G q(M),i 6 {1, . . . ,m}} < e ■ i{f)~ l (23) 

where m = 1 and I\ = I for case (ii). 

Let fM be absolutely continuous with respect to A and denote the density of p with /. Let 

For any q G Q let 

N q = {a G q(R) : /i(g -1 (a)) > 0}. 

In the case a < we need to control in our proofs the cardinality of the codebook of any quantizer 
whose entropy is less than or equal to the rate constraint R. To this end, for R > 0, we define 

Ur = {q G Q c : Hfjftq) < R, Ce R < cavd(N q ) < e R }. 
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In addition, we will have to control the difference between the rate constraint and the entropy of the 
quantizer. Thus, for a G (—00, 0), arbitrary constant k > 0, and R > log( 2 we define 



l/(l-a) - 



The next lemma is proved in Appendix B 



Lemma 5.4. Let p be absolutely continuous with respect to A having density f. Assume that supp(/^) is 
a compact interval and < i( f ) < s(f) < 00. For every a G [—00, 0] and R > we have 

D°KR)=w£{D tl (q):q€H R }. (25) 

If a G (—00, 0) am/ > log( 2 q -1 ), then 

D«(R) = inf{^(g) : 9 € /C fl (C)}. (26) 

We let U (I) denote the uniform distribution on a bounded interval I C M with positive length. 
Let m > 2 and let Ji, . . . , I m be a partition of I into m intervals of equal length diam(/)/m. Let 
si, . . . , s m € (0, l) m with Sj = 1 and assume the source distribution is of the form p, = YliLi s iU(h 
Of special interest in our proofs are the codecells which are straddling the intervals Ij. Hence we define 
for any quantizer q G Q the sets 

m 

A(q) = \J{aeq(R):\(q- 1 (a)\I i )=0} and S(q) = q(R) \ A(q). (27) 
i=i 

In the proof of our main result we have to ensure that the contribution of the straddling cells to the overall 
entropy of the quantizer can be (asymptotically) neglected. For a < this is the case if it suffices to 
consider only quantizers with the property that the length of each straddling cell is at least as large as a 
certain (fixed) constant times the length of the smallest non-straddling cell. Exactly this is ensured by the 



following lemma which sharpens Lemma 1541 The proof is given in Appendix B Recall the definition 
d23J of the constant C and let k £ (0, C). For R > log( 21 ^ -1 ) let 

Gr = Gr(k) = {q G JCr(k) : 2inf{diam(?- 1 (a) H I) : a G 5(g)} 
> inf{diam(g~ 1 (a)) : a G A(q)}}. 

Lemma 5.5. Assume that /j, = Y^7=i s iU{h) is a piecewise uniform distribution as specified above. 
Let r > 1 and k G (0, C). Then for every a G (— 00, 0) there is an i?o( K ) > such that for every 
R > Rq(k), 

D*(R) = w£{D lt (q):q€g R ( K )}. 
A bijective mapping T : R — > R is called a similarity transformation if there exists c G (0, 00), 
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the scaling number, such that \Tx — Ty\ = c\x — y\ for every x, y € R. The last result of this section 
describes how the optimal quantization error scales under a similarity transformation. For a = the 
reader is also referred to (9l Lemma 3.2]. Let us denote by 

C^R) = {qeQ c : D,{q) = D«(R)} 

the set of all optimal quantizers in Q c for p under Renyi entropy constraint R of order a. 

Lemma 5.6. Let a € [—00, 00] and T : R — > R be a similarity transformation with scaling number 
c > 0. Then for any R>0we have 

D« oT _ 1 (R) = c r D«(R). 

Moreover, 

C^ oT - 1 (R) = {To q oT- 1 : q eC^R)}. 

Proof. The lemma follows because for any q € Q we have q := T o q o T _1 € Q, H a oT _i (q) = (q), 
and D^oT-i (q) = c r D^{q) (also, g € Q c iff q G Q c ). See also [19, Lemma 2.4] where a > and r > 1 
are considered, but the same proof clearly works for all a < and r > 0. □ 

6 Inequalities for mixture distributions 

In this section we provide upper and lower bounds for the optimal quantization error of mixture distribu- 
tions in terms of the optimal quantizer performance for the component distributions. Proofs are given in 
Appendix C 

Definition 6.1. Let m > 2 and A%, . . . , A m be measurable sets which are pairwise disjoint. The dis- 
tribution fx is called m— divisible with respect to (A\, . . . , A m ) if fJ,(Ai) > Ofor all i = 1, . . . ,m and 
M(U£i^) = I- 

For any measurable A C R with p,{A) > we let fi(-\A) denote the conditional probability of p, 
with respect to A, i.e., p(B\A) = [i(B n A)/ p{A) for all measurable B C R. If //is m— divisible, then 
we write pi = p(-\Ai). 

Proposition 6.2. Let R > 0, a £ [0, 00) \ {1}, and m > 2. Assume that p is m— divisible with partition 
(Ai, . . . , A m ). Moreover assume, that J \x\ r dpi(x) < 00 for every i = 1, . . . , m. Let Ri, . . . , R m € 
[0, 00). Letting S{ = p(Aj), we have 

m 

D*{R) < ^SiD^. (Ri) 
i=i 
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if either one of the following inequalities holds: 

(m \ 

< (l-a)R ifae[0,l), (28) 

logf^^ 1 -^) > (l-a)R ifae(l,oo). (29) 



•,i=i 



Recall the definition (flOl ) of ai and 02- Let m > 2 and si, . . . , s m G (0, l) m with s i = 1- F° r 
every i G {1, . . . , m} and a G [0, r + 1) \ {1} let 
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*< = 'J IE*? J • (30) 

Lemma 6.3. Le? m > 2. Le? fte non-atomic and m— divisible with respect to (A\, . . . , A m ). Assume 
f \x\ r d(ii(x) < 00 for all i = 1, . . . , m. Let iq G {1, . . . , m} with fi(Ai ) = s = max{fi(Ai) : i = 
1, . . . ,m}. If a G [0, 1), then 

\immfe rR D"(R) > s aia2 lim inf e rR D%. (R). (31) 

Let Si = fi(Ai) and assume a G [0, r + 1) \ {1}. Then we have 

m 

lim sup e rR D« (R) < Si tT r lim sup e rR D«. (R) . (32) 

7 Proof of main result 

Recall that C7 (J) denotes the uniform distribution on a bounded interval / with positive length. First we 
show that the optimal quantizer performance for U(I) is the same for all negative a. 

Lemma 7.1. Let —00 < a < b < 00. For every R > am/ a < 0, we have 

D U{[a,b])( R ) = - D f/([a,f)])(- R )- 

Proof. Note that by Lemma [5761 it suffices to consider the case [a,b] = [0, 1]. Since D ^ (R) is 
nonincreasing in a by Lemma 1231 it suffices to prove the assertion for a = —00. Let R > and assume 
q G Q satisfies H~°°{q) < R. Setting 

iV, = {a G g(R) : v(q~\a)) > 0} 
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this condition is equivalent to 



p = min{^(g _1 (a)) : a E N q } > exp(-R). (33) 

Let [x\ denote the largest integer less than or equal to x € R. Using 1 > card(iV g ) • p we get 

card(TVg) < [exp(R)\, (34) 

which is equivalent to H®(q) < R. From, e.g., ||9j Example 5.5] we know that only the quantizer 
g G Q which partitions the unit interval into [exp(i?)J intervals of equal length with their midpoints 
as quantization points, attains the optimal error, i.e., IV([o,i]) (#) = ^([oi])^)' ^ ut tms 1 uant i zer 
satisfies conditions (l33l) and (l34l) simultaneously. Hence, D u ^ ^(g) = -q\(R), which yields the 

assertion. □ 

Next we determine the exact behavior of D^,, Q ^(R) for large R. For a = 1 the following result is 
from [13]. For the case a = the reader is referred, for example, to [9, Example 5.5]. 

Proposition 7.2. Let r > 1 and R > 0. Let — oo < a < b < oo. Then the following hold: 

(i) If a € [0,r + 1), an optimal quantizer always exists for U([a, b]), i.e., we can find a q E Q with 

H U{[aM)^ < Rand D U([a,b])(<l) = D U([a,b])^- 

(ii) Suppose a S [0, r + 1) and let n € N Z?e smc/j ?/zaf i? € (log(n), log(ra + 1)]. Then the restriction 
to [a, b] of the quantizer q in (i) has (n + 1) interval cells, n of which are of equal lengths and one 
having length less than or equal to that of the others. Ifa>0, then q meets the entropy constraint 
with equality, i.e., H^^ ab ^(q) = R. 

(Hi) For all a € [—00, r + 1), we have 

lim o e rR D» m]) (R) = C(r). (35) 

Proof. Assertions (i) and (ii) follow directly from [ 19, Thm. 3.1] by noting that in view of Lemma l5.6l it 
suffices to consider the case [a, b] = [0, 1] 

To prove (iii) first we note that by Lemma rTTTl the limit 051 ) holds for all a € [—00, 0) since it holds 
for a = 0. Thus we need only concentrate on the case a G (0,r + 1) \ {1}. Applying [19, Thm. 3.1] we 
obtain 

DZ (m) (log(n)) = C(r)n- r 
for every reel Now let R > and n R € N, such that log(n^) < R < \og(n R + 1). We get 



n r R C(r)(n R + 1) r = n r R D% m]) (log(n R + 1)) 

} U([o,l])( R ) ^ ( n R + 1 ) r - D U([o,i]) 



< e rR D% m]) {R) < (n R + 1)^^(1^) = (n R + l) r C(r)n R r . 
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Letting R -> oo yields 435j for a G (0, r + 1) \ {1}. 



□ 



Proof of Theorem We divide the proof into four main steps. In step 1 we begin by proving a (sharp) 
asymptotic lower bound on the optimal quantization error for any distribution with a density that is 
piecewise constant on a finite number of intervals of equal lengths. In step 2 we generalize the lower 
bound of step 1 to any density whose support is a compact interval on which it is bounded away from 
zero. Together with a matching upper bound based on the companding result Corollary 14.1 II this will 
finish the proof for a G (— oo,0). In step 3 we show that the lower bound holds for all distributions 
subject to our restrictions and apply again the companding upper bound to finish the proof for a G (0, 1). 
Step 4 treats the remaining a = — oo case and thus completes the proof. 

Throughout we assume w.l.o.g. that R > Rq{C /2) where is C defined in (1241 and Rq(C/2) is from 
Lemma 1531 
Step 1. 

Let M be a compact interval of positive length and let m > 2 and a G (— oo, 1) \ {0}. Assume that 
fi = YliLi s iU {Ai), where the Ai are disjoint intervals of equal length / = l(Ai) = X(M)/m that form 
a partition of M. We assume > for all i = 1, . . . , m. Thus Y^Li s « = 1 an d 

i 

For a G (-oo, 1) \ {0} define t { = s\ /a2 feyLi s f) > » = *> ■ ■ ■ > m as in &■ Let 

R > max{0, max{— log(ij) : i = 1, . . . , m}} 

and define 

Ri = R + log(ti) > 0. 

From Proposition 17.21 we deduce 

e rR D« m]) ( Rl ) = (e R - R >ye rR >D« m]) ( Rl ) 
-> t^ r C(r) as R -> oo. 

A simple calculation shows 




19 



Now, according to Lemma 1531 there exist functions e : (0, oo) — > (0, oo) and 5 : (0, oo) — > (0, oo) such 
that for every R > and quantizer q G Q with H%(q) < R and \D%(R) - D^(q)\ < 5(R) we have 

max{ / u(^ 1 (a)) : a £ g(R)} < e(i?) (37) 

where e(J2) — > as — > oo. Moreover, 

max{diam(Aj f1g" 1 (a)) : a G g(R),i G {1, . . . ,m}} < — f^fG^) ^_ ^ 

min{sj : i = 1, . . . , m} 

Now, again, let R> Rq and 7 > 0. According to Lemma 1231 let G Q be a quantizer whose codecells 
with positive /x— mass are intervals, satisfying H^(q R ) < R and 

\D«(R) - D^{q R )\ < min( 7e - ri? , ( 5(i?)). (39) 

Hence, q R satisfies also the relations (|37T > and (|38T >. In view of Lemma 1531 let us assume w.l.o.g. that 
Qr S Qr if ol < 0. Now let i £ {1, ... , m} and 

/i(g«) = {a G ?Ji (R) : A(^ 1 (a) \ A t ) = 0} 

and 

A i,<in = U ^(a)- 

With 

Mqr) = {a g g*(K) \ ii(gji) : m(A n g^W) > 0} 

we obtain from (1381 ) that 

lim sup{diam(Aj n qp 1 (a)) : a G Ji(qR)} = 0. 

Every point of Ji{qn) is a codepoints of a codecell which is straddling the boundary of Ai and is not 
\i — a.s. contained in A{. Hence {Ai n q~^{a) : a G Ji(gfl)} consists of at most two intervals and we get 

lim diam(^4i 9 „) = diam(j4j) = I. (40) 

R-^rco 

We compute 

m . 

Dh(<1r) = ^2 s i rl \x - q R (x)\ r d\(x) 
i=i jA i 



> 



m ,. 

J2 Sirl \x - q R (x)\ r d\(x). (41) 

1=1 J Ai <lR 
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Let 



Ri, qR - Hjj( A t )(qr) 



\aeii(<Jfl) / 



a 
a — 1 



(42) 



where J7(Aj (?fl )(q R 1 (a)) is the measure of the cell Q , ^ 1 (a) under the uniform distribution on Ai AR . Then 
using Lemma [5^61 and (|4~TT) we obtain 



- E Si * lD U(A l , qR )( R i,q R ) d[am ( A i,q R ) 
i=i 

m 

= E s * rl ^([o,i])(^^) diam (^^) 1+r - ( 43 ) 



8=1 



Now pick a sequence (L n ) of non-negative real numbers, such that L n — > oo, 

e rL "D«(L n ) -> liminf e rR D%(R), (44) 

^ if,— S>oo ^ 

and 

— p > Vi £ 10, col, £ = 1, . . . , m (45) 

as n — y oo. Because we want to determine a lower bound for the optimal quantization error, us- 
ing Proposition 17.21 (i) we can assume w.l.o.g. that q^ n is Ri,q L —optimal for U{Ai ALn ), i.e., that 
D U(A ia )( R i,iL n ) = D U(A t . q , )(<?lJ- By Proposition E2](ii) the quantizer q Ln divides A iiqLn into 
(k + l)-intervals with Ri, qLn £ (log (A;), log (A; + 1)] where at least k intervals are of equal length. 

We next prove that Ri, QLn — > oo as n — > oo. Assume to the contrary that {Ri, qLn ) n eN is bounded. 
Then k = k(Ri jqL ) will also be bounded. Thus let ^eN such that k G {1, . . . , ko} for every n € N. 
Together with (l40l we deduce 

liminf DJqzJ > liminf \x - q Ln (x)\ r d/j,(x) 

n— too n— ¥oo I a 

diamfA; ) 
Si I 2 fc+1 

> liminf— k -2 / x r d\(x) 

> C(r) Si r min{ + fc +1 : fc g {l,...,fc }| > 0. 
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But this contradicts (cf. Corollary 15.21 ) 



lim sup < limsup(^(L n ) + 7 e- rL ") = 0. 



n— >oo 



Thus we obtain that R-i, qLn — >• oo as n — > oo for all i € {1, . . . , m}. Proposition 17 . 2 1 yields 

Km e r ^^«^ ([0il]) ( J R^ L J = C(r), i = 1, . . . ,m. (46) 
Because 7 > was arbitrary we obtain 

liminf e rR D°(R) = lim e rLn D°!(L n ) = lim e rLn Djq Ln ) 

m 
8=1 

m 

= C(r)^ Si ^r (47) 
i=i 

where the first equality holds by (l44l . the second by d39l ), the inequality follows from (l43l >. and the 
third equality follows from (l40l) . (1431) . and (l46l ). In the last expression, = if Uj = 00. The case 
Vi = cannot occur because otherwise the right hand side of d47l is not finite, which would contradict 
the assertion of Proposition 15. II Recall that {Ai n : a € Ji(qit)} contains at most two intervals 

for every i and n G N. Now assume that a G (0, 1). In this case, since by © we can assume w.l.o.g. that 
H ^QL n ) = L n , we obtain 



1 < 5i(L n) ^,q Ln ) :-- 



e £n(l-«) 



< 



E™1 Ea G / l(gi J^fe!( a )) a 
E£l EagJ t ( gL J ^(gL!( Q ))" + Eagg £n (R)\U^ 1 J J ( t?L , i ) KQlI^Y 

EZi Ea e / l(gL j/^fe!(«)) Q 

E™ 1 Eoeii( 9£n ) M9Z!( a )) Q + (m + 1) sup o6?£n(K) ^(g^(a)) a 



1 + 



E£i Eae/^jM^fa)) 
(m + 1) sup ae (R) ^{qll{a)) a 



From (1421 and lim^^ Ri, qL = 00 we deduce 



Km ^ £ /*(<£»)" = °o. 

-n. — ^nn ^ — * ^ — * n 

Thus we get 



n— >oo ' 

i=l a&h(q Ln ) 



lim 5i(L n ,/i,g L J = 1. (48) 
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Using Lemma ID7T1 in | Appendix D| we recognize that the limit relation (|48T ) also holds for a < 0. Conse- 
quently, we deduce together with (1401 and ((42J) for every a G (—00, 1) \ {0} that 



-L„)(l-a) 

"1 ----- 

n— >oo 

i=l 



\"sfvj- a = Vsf lim e ( 

j=l 



i=l aeIi{q Ln ) 



Moreover we obtain from (08]) and (|42l) that Vi < 00 for every i = 1, . . . , m. Since 2~2iLi s f v i ° = 1> 
we can apply Lemma [DT2l ( |4"7T ), and ( f36b to obtain for q G (—00, 1) \ {0} that 



liminf e rR D^(R) > C(r) f>^T = C(0 f / 
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Step 2. 

Now let us assume that the support of fj, is a compact interval M C K and that / is continuous on 
M. Again, let a G (-00, 1) \ {0}. Let i(f) = min{/(x) : x G M} resp. s(f) = max{/(x) : x G M}. 
Clearly, s(f) < 00. Let us assume that 

•(/) > 0. (49) 

Let I = X(M). For k G N partition M into intervals {^4j : % = 1, . . . , k} of common length A(vlj) = 
Set 

j=l 

and 

k 

_ djik_ _ sr^ HyAj) 

Jk ~ dX ~^\{Ai) Av 

i=i 

The continuity of / implies that converges point wise to / as k — > 00. In view of d49l and due to 

i(f) = min{/(x) : x G M} < min{/fc(x) : x G M} 

< max{/ fe (x) : x G M} < max{f(x) : x G M} = s(f) < 00 

for every k G N, dominated convergence implies 

lim / f^dX = [ f a 'dX. (50) 

fe ^°° Jm Jm 

Moreover step 1 yields 

lim. inf e rR D« k (R) > C(r) (J /«> dx\ \ (51) 
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Now let R > max(l, Rq). Let 5 > and q R be a quantizer with \D^(R) — D^qn)] < 5e~ rR and 
Hu(Qr) < -R- I n addition, we assume w.l.o.g. (cf. Lemma [5~4l resp. Lemma [23T > that q R € Kr{C) if 
a < and i? is large enough, and H^(q R ) = R if a > 0. For i = 1, . . . , k let 

< Ci }k = mm{f(x) : x € Ai} < t i}k = max{/(x) : x £ A4} < 00 

and 

< c k = min < — 2 — : i = 1, . . . , k > . 

I U,k J 

For every a £ <7.r(M) we have 

c k n k {q^-(a)) < niq^ia)) < c^ 1 fi k {q R l {a)) . 
and because / is uniformly continuous, in view of d49t , we have 

lim c k = 1. (52) 

fc— >oo 

We obtain from the definitions of H^{q R ) and c k that 

< e fl?(9K)-H^(9«) < max^.cpj =: ^ (53) 
where v k — > 1 as k — > 00. Again from the uniform continuity of / we deduce 

lim ||/- Alloc = (54) 

k — ^00 

with 

11/ - Alloc = max{|/(x) - f k (x)\ : x E M}. 
In view of Proposition 15. 1 1 there exists an mo > 0, such that for all R > 1 

D°(R) < m e- rR . (55) 

By the choice of q R we have 

e rR D^(R)>e rR D^q R )-5. (56) 

Thus (gD yields 



llVgiO-^tefl)! < / |x- 9fl (x)| r |/(x)-/fc(x)|dA(x) 

< \\f ~ fk\\oo-T77 I \x -q R (x)\ r f{x)d\(x) 
H/J -/at 
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< 



/ k I oo" 



m + 5 _ rR 
Kf) 6 ' 

Hence, (1561 ) gives 

e rR D«(R) 

> e rR (D^ k (q R ) - \D,{q R ) - D, k (q R )\) - S 

> e^D, k (q R )-\\f-f k \\^^±^-5 

= e r^^)e rH ^D, k { qR ) - ||/ - /*||»^ - 5. 

-ilZ-AlU^-^. (57) 

*(/) 

Due to the choice of g_R there exists a function g : (0, oo) i-> (0, oo) with e R ~ H %( qR *> < g(it!) and 
— > 1 as i? — > oo. Equation (|53T > implies 

\H% k (q R ) -R\< \H« k (q R ) - H«{q R )\ + \H«(q R ) -R\<\ \og(v k )\ + \ log(g(R))\. (58) 

Clearly, inequality (|58T > yields Hindoo H^ k (q R ) = oo. Applying relations (|58T > and (f5TT > to (fSTb we 
deduce 

liminf e rR D%(R) > e - r ^^C(r) ( [ dxY - \\f - MU^^ - S. 
By letting A; — > oo and noting that 5 > is arbitrary we obtain from (|50T >. (l52l) and (l54l) that 

liminf e rR D™(R) > C{r) ( ( f ai dx] . (59) 

R -+°° \Jm ) 

Next we show a matching upper bound for a S (— oo,0). The assumptions on / allow us to use 
Corollary 14. 1 1 1 showing the existence of a sequence of companding quantizers (q^) = (Q/*,n) such 
that a2 

lim e rH ^ (qN) DJq N ) < C(r) ( [ f 1 dx] * . (60) 

Let Rn = H^(qN) and note that Proposition 14.81 implies 

lim Rn = oo, lim (Rjy — Rn-i) = 0. (61) 
Let R > be arbitrary and let n = max{iV : Rjy < R}- Then i? n < R < R n +\ and since D^(R) is a 
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nonincreasing function of R 



e rR D«(R) < e rRn+1 D^R n ) < e r (^+i-R n ) e rR n D ^ qN ), 
This, ([SB, and dgfl yield 

Wiasuv e rR D a {R) < C(r) ( [ / ai dx) . 

R^oo ' \Jm J 

Together with (1591 ) this completes the proof for the case a G (—00, 0). 
Step 3. 

Now let [i be arbitrary, but satisfying all assumptions of the theorem. Let a G (0, 1). For k, I G N let 

ii = (-«>,-*), / 2 = [-fc ) fc]nr 1 ([iA,J]) 1 = 00) 

and 

/ 4 =M\(/ 1 U/ 2 U/ 3 ). 

Because / is bounded and weakly unimodal we can pick k G N such that n{I 2 ) > 0, = 
00)), 1/7 < Z (see Definition 1521. and 

|i(/ 2 ) = max{/i(/ i ) :j G {1,2,3,4}} 

for every k > ko and I > ko. Note that I2 is a compact interval. Now let min(/c, I) > ko- Let us first 
assume that fi(Ii) > for every i = 1,2,3,4. Consider the decomposition [a = Yli=i M-^OM'I-^)- 
Lemma 16.31 yields 

liminf e rR D°(R) > ^{h) 1 ^^ liminf e rR D°! (4I JR). (62) 

By construction, i(/i(/2) _1 /l/ 2 ) > 0, s(^(/2) _1 /l/ 2 ) < 00 > an d M-^) -1 /!^ * s supported by a com- 
pact interval. Thus we can apply the results of step 2. Together with the definition of a\ and a 2 we 
deduce from (|59]> and d62j that 

liminf e rR DZ(R) > ^h) 1 ^ C (r) (»(l 2 y l f) ai dxj* 

l — a-\-ar _ _ f [ \ - 



/ 2 



, a2 



C(r) [ f^dX . (63) 
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Due to a\ > and by monotone convergence we obtain 



A:— »oo I— >oo 



lim lim / f ai dX = / f ai dX. (64) 



a 2 



Thus we get from (1631) that 

lrminf e rR D«(R) > C{r) (J f ai dX^j . (65) 

The case min{/i(/i), ^(h), ^{h)} = can be treated similarly. 

To show the matching upper bound, note that since 1 jl < f < I on I 2 , we can directly apply 
Corollary 14. lll to '■= and its density f 2 := ^{I^ 1 /l/ 2 to show the existence of a sequence 

of companding quantizers (gAr) = (Qj^n) sucn that 

Y^e rHa ^D^ 2 {q N ) < C(r) (J dX^ ** 
Thus by the same argument as in the previous step 

lim sup e**!^ (R) < C(r)(lf?dx) 

= C(r)fi(I 2 )- a ^ Qf f^dxj 2 . (66) 
Again from Lemma [631 we obtain for a G (0, 1) the upper bound 

limsupe r/? Z)"(i?) < C(r)fi(I 2 ) 1 ' aia H 2 r ( [ f ai dx) % 

R-t-oo ' \Jl 2 J 

+ 3 max ^h)t- r lim sup e rR D« m (R). 

*= 1,3,4 R^oo ^ 1 ' 

Using Proposition 15. 1 1 we get a K > independent of k,l such that 

limsupe rfl £>?, | n (i?) < iv-^(/i)" r/(r+,5) . 

for i G {1, 3, 4}. Letting I, k tend to infinity we obtain by the definition of U that lim^fc-s.00 t^ r = 1, 
resp. lim^fc^oo t^ r = 0, i = 1, 3, 4. Using (l64l and (l66l ) we get 

«2 



limsupe ri? ^(i?) < C(r) (V / ai dA^ 



which, together with the lower bound ( [651 completes the proof for the case a G (0,1). 
Step 4. 
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Let a = — oo and f} € (—00, 0). Fix a\ = ai(j3) and a% = a^{P)- From Lemma 1231 we deduce 



liminfe rjR J D,T 00 (i?) > lim e rR D^ t 



R— >oo i?— >>00 




supp(Ai) 



r 1 ^ 



) 



Since the integral on the right hand side converges to f f 1 T ' dX as /3 — > — 00, we obtain 



liminfe r ^L>- 00 (i?) > C(r) 

ft — irv-t " 



_R— s>oo 



supp(/i) 



/ x - r dA. 



The proof is finished by noting that Corollary 14.1 11 and an argument identical to the one used in step 2 



8 Concluding remarks 

We have determined the sharp distortion asymptotics for optimal scalar quantization with Renyi entropy 
constraint for values a € [—00, 0) U (0, 1) of the order parameter. Our results, together with the classical 
a = and a = 1 cases, and the recent result [18] for a S [r + l,oo], leave only open the case 
a 6 (1, 1 + r) for which non-matching upper and lower bounds are known to date (cf. [18]). We note 
that the upper bound provided by optimal companding in Corollary 14. 1 1 1 also holds for a € (1, 1 + r). 
Based on this, we conjecture that our main result is also valid for this remaining range of the a parameter. 

Apart from the question of high-rate asymptotics, it remains open if optimal quantizers exist for all 
a G [— 00, 1 + r]. The non-existence of optimal quantizers in case of a > 1 + r has already been shown 
in lEl. Looking at our main result, it is obvious that the integrals on the right hand sides of (fTTT) and (fl2l 
are not finite in general if fi has unbounded support. It needs further research to determine the exact high- 
rate error asymptotics for certain classes of source distributions with unbounded support and a < 0. Of 
special interest is the question whether companding quantizers with point density /* are still asymptot- 
ically optimal for source densities with unbounded support. The definition of /* needs the integrability 
of f 1 ^ 2 in order to guarantee a finite number of quantization points for the (asymptotically optimal) 
companding quantizer. Nevertheless, the right hand side of (fTTT) is defined only when f ai is integrable. 
It remains an open problem if (fTTT> still holds for some a € (— oo,l + r)\{l} and distributions where f ai 
is integrable but f 1 / - 2 is not. Such an example, if it exists, would show that the companding approach is 
not always applicable to generate asymptotically optimal quantizers, but the known asymptotics (fTTT > are 
still in force. Another interesting open question is whether the non-integrability of f 1 ' 02 always implies 
the non-existence of optimal quantizers with a finite codebook. 

A careful reading of the proofs shows that many arguments can be straightforwardly generalized to 
the d-dimensional case and rth power distortion based on some norm on Mr. For a G [—00, 1) and under 
appropriate conditions we conjecture that 



provide a matching upper bound. 



□ 
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where X d is the d-dimensional Lebesgue measure, 



l-q + q a l- a + 

«i - -j — — , a 2 - — 

1 — ol + ^ 1 — a 

and C(r, d) is a positive constant that depends only on r, d, and the underlying norm. 

However, some important steps in our proofs are definitely restricted to the scalar case, e.g., equation 
(1231 in Lemma 15.31 which yields (l40l ). One of the key problems concerns the first step of the proof 
of Theorem 13.41 In higher dimensions one has to control the contribution to distortion and entropy of 
cells straddling the common boundary of at least two touching cubes in the support of \i. The "firewall" 
construction used in case of a = (see (H p.87]) does not seem to work in the general case. For a/0 
it seems to be very hard to control the entropy of the quantizer when adding or changing codecells and 
codepoints in a certain region. In order to progress in this direction, one would certainly need more 
refined knowledge about the codecell geometry of (asymptotically) optimal quantizers. Even in the 
case a = little is known on this subject (results in [29] highlight the difficulty of the problem). As 
already mentioned in the introduction, the methods used for the case a = 1 are also not applicable to 
the general case because they rely on the special functional form of the Shannon entropy. It appears that 
generalization to higher dimensions would necessitate the development of isodiametric inequalities for 
the (bounded) codecells of asymptotically optimal quantizers. 

Appendix A 

Proof of Lemma \2~3\ To show (@]), let q € Q be such that Hjf(q) < R and assume /3 < a. It is easy 
to check that -^H^(q) < on (— oo, 0) U (0, 1) U (1, oo), and thus the mapping 7 (->• H^(q) is non- 
increasing on these intervals. In view of the continuity of H^(q) at a 6 {0, 1} (see (Remark |2.2f a)) we 
deduce that H^{q) < H^(q). Now the assertion follows from Definition ((3]). 

Equation (f5]) of the second statement follows directly from the more general results Theorem 3.2 and 
Proposition 4.2 in ||20l . For ©, we refer to |fl9l Proposition 2. l.(i)]. □ 

Proof of Proposition \4.2\ We proceed in several steps. 

1. Since G is increasing, it has a derivative & a.e. (by convention we set G'{x) = if G is not 
differentiable at x). Also, note that G and G are strictly increasing on I, resp. on [0, 1], and G is 
Lipschitz with constant (ess inf/ g) -1 and thus absolutely continuous. Since G'{x) = l/g(G(x)) a.e. on 
(0, 1), we obtain 

^d\ = j{G\G(x))Yd^x). 



2. Next we prove 



lim / (m N (G(x))) r dfi(x) = f (G'(G(x))) r d/j,(x), (67) 

J J 
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where is the piecewise constant function defined by m^{x) = N S[{%-i)/n i/N) ^ x e [(* ~~ 
1)/N,i/N), i = 1, . . . , N and mjv(z) = otherwise. 

Lebesgue's differentiation theorem (see, e.g., Q Thm. 6.2.3]) implies that rriN(x) — >■ G'(a;) as 
A 7 — > oo a.e. on (0, 1). Also, from G'(x) = l/g(G(x)) we deduce 

mjv(-) < max\N I — J dA(x) : 1 < i < N 1 

I J[(i-i)/N,i/N) g(G(x)) J 

< (essinfj g) -1 . 



Thus, the dominated convergence theorem yields (I67I ). 

J. Let Q g ,N be the quantizer with the same codecells as Q 9j n but with the midpoints of the codecells as 
quantization points: 

Q 9 ,n(x) = \(G{i/N) + G((i - l)/N)) if x G I i;N , i = 1,... ,N. 
We will show that 

lirn^N 1 - J \x-Q g , N (x)\ r f N (x)dX(x) = C(r) J -L dX, 

where is the piecewise constant density defined by /at (a?) = x(L N ) fj. N f d\ if x G Ii,N> * = 
1, . . . , A 7 ' and /at(x) = otherwise. 
A simple calculation shows 

A^y |x-Q 5iJV (x)|77v(x) d\(x) 

= N r Y,c(r)f N Mjj)) m,N)) r+1 

1=1 ^ ^ ' ' 

N 

= C(r)N r J2 /(x)(J\rW(G(s))) r dA(a;) 

i=l Jh,N 

= C(r) [ m N (G(x)) r f(x)dX(x). 



Now the assertion follows from steps 1 and 2. 
4. Next we show that 



lim N r D IM (Q g>N ) = C(r) [ L dX. 



For any i G {1, . . . , N} and x G I^n we have 



N r \x-Q g>N (x)\ r <iV(A(/ ii7V )) r = iVM / G'dX) =m N (G(x)y. 

\J((i-l)/N,i/N) J 
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Therefore 



N r / \x-Q g , N (x)\ r f(x)dX(x)-N r j \x-Q g)N {x)\ r f N (x)d\{x) 

< J m N (G(x)) r \f(x) - f N (x)\dX(x) 

< (essinf/5-) -1 / \f(x) - f N (x)\d\(x). 



By Lebesgue's differentiation theorem we have /jv — > f a.e., and now Scheffe's theorem (4) Thm. 16.1 1] 
implies 



Hence, 



lim / \f-f N \d\ = 0. 



lim N r D^Q g , N ) = lim N r [ \x - Q 9jN (x)\ r f N (x) d\(x), 

iv— >oo N— s-oo J 



where the right hand side is equal to C{r) f dX from step 3. 
5. In view of step 4, to prove relation ([TBI ) it suffices to show that 



lim N r D^Q 9iN ) = lim N r D^Q g , N ) 



(68) 



Applying the mean value theorem of differentiation (if r > 1) or by the triangle inequality (if r = 1), 
we have for each i € {!,..., N} and x G I^jv, 



N r \\x-Q g;N (x)\ r -\x-Q g;N (x)\ r \ 

< N r \Q gtN (x) - QaMxMKkN))^ 1 - 



(69) 



Further, note that the definitions of m^, Q 9) n, Qg,N also yield 

N r \\x - Q g , N (x)\ r -\x- Q g>N (x)\ r \ 

< rN r (X(I hN )Y =r-m N (G(x)Y 



(70) 



Let l} N and if N denote the partition of J» jv into two intervals of equal length A(/j i 7v)/2. Let 
j = j(x) G {1, 2} be such that x G P iN . Letting a = inf(Ii jv) and b = sup(Ij i Ar) we obtain by the 
absolute continuity of G 

r( a + fc )/ 2 „ 

G((o + 6)/2) = G(a) + G' dX 

J a 

and 

gw±M = G M+ /'- +t)/2 M^w rfA . 

2 .L b - a 
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Thus we get 

G(a) + G(b) 



\Qg,N(x) - Qg,N{x)\ 



G{{a + b)/2) 

< X(ll N )\L(x,N)\ (71) 



where 



XilKAJp b-a 

i,n i ji i,N 

' ' G" dA - — ; / & dX 



if x £ i^jy, i = 1, . . . , A r . In view of (l69l ) and (1711 ) we deduce 

iV r ||z - Q 9 ,jv(x)| r - \x - Q 9 ,N(x)\ r \ < rN r X(ll N )\L(x,N)\(X(I i , N )) r - 1 

< r-m N (G(x)) r \L(x,N)\. 

Lebesgue's differentiation theorem yields liniAr^oo L(x, N) = a.e. Hence 

lim N r I \x - Q g ,N{x) r - \x - Q g , N (x) T I = a.e. (72) 

N->oo 



Due to the relations (1721 . (1701 ) and together with step 2, we can apply the generalized dominated conver- 
gence theorem (28] Chapter 1 1 .4] to obtain (l68l □ 

Lemma A.l. Le? fibe a probability distribution which is absolutely continuous with respect to X and let 
f denote its density. Let G be a compressor for p, with point density g. If {g = 0} C {/ = 0}, then 
p o is absolutely continuous with respect to X. Also 

= -7XTV, l { 9 >o}{G(y)) a.e. on (0, 1) (73) 

and p o G^ 1 has the density 

fa{y) = f(G(y))G'(y) = l^Ml {g>0} (G(y)) a.e. on (0, 1). (74) 

Proof. In order to prove that p o G^ 1 is absolutely continuous let us make the key observation that, 
although G is in general not invertible, we have 

G{G{x)) = x p-a.e. x G M (75) 

Indeed, by the definition of G and due to {g = 0} C {/ = 0} there exists a measurable set Ac C R 
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such that G is differentiable on Aq, /j-(Ag) = 1, and 



G'(x) = g(x) G (0, oo) for every x G Aq. 
Hence G is locally invertible at x € Aq, so G(G(x)) = x which proves (f75l Moreover, 

&{G{x)) = l/g(x) for every iei G (76) 

which proves d73l . 

G is strictly increasing (and thus one-to-one) and maps (0, 1) onto G((0, 1)). Thus, together with 
(|75T > we obtain for every Borel measurable BcR that 

H o G~ l (B) = n({x : G(G(*)) e G(B)}) = »(G(B)). (77) 

If Z7([0, 1]) denotes the uniform distribution on [0, 1] we obtain again from (1751 ) that 

U([0, 1]) o G^d-oo, x}) = G(x) for a.e. iGl. 

Thus, f7([0, 1]) o G" 1 = gX. Now let B C (0, 1) be Borel measurable and X(B) = 0. This implies 
U([0, l])oG _1 (G(B)) = 0. Because //is absolutely continuous with respect to gX we obtain fj,(G(B)) = 
0. Hence, (1771 ) implies fj, o G^ 1 (B) = showing that // o G^ 1 (B) is absolutely continuous with respect 
to A. In order to prove d74l ) let [a, 6] C (0, 1). In view of (1771 ) and from the definition of G we obtain 

r G(b) 

Li(G-\[a,b])) = / /dA. (78) 

JG{a) 

From d76l ) we deduce 

= -7^ 1 { 9 >0}(G{V)) a.e. on (0, 1). 

Because [i o G _1 is absolutely continuous with respect to A its cumulative distribution function is ab- 
solutely continuous and, therefore, differentiable a.e. Applying the chain rule for the Lebesgue integral 
(see [30, Corollary 4]) we obtain 

G(b) rb 

f(x)dX(x)= f(G(y))G'(y)dX(y). (79) 



'G(a) 

Now, (|78T > and d79l prove the first equation in d74l) . The second equality in d74l follows from (f73l . □ 



Proof of Lemma \4.6\ 

1. a £ (0, oo). For this range of a the result goes back to Renyi ll27l 1 1 §] who stated it with somewhat 
less generality. Csiszar [8, Thm. 2] gives a more general form of the result that implies our statement. 
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Clearly, 

mf{n(q£ M (a)) : a G q a ,mW} 
limml > ess ml mj- (80) 

Now let e > and define N £ = {x : f(x) < essinf M f + e} n M. Hence, X(N £ ) > 0. By Lebesgue's 
differentiation theorem we can find an x G N £ such that \i is differentiable at x with f(x) = ^fr(x). 
Moreover a Ao(e) > exists, such that for every A < Ao a b G <?a,m(K) can be found with x G 

9a?m( 6 ) and 

KgA?Agj^ 

A 

Because e is arbitrary we obtain 



< ess inf m/ + 2e. 



inf{/i(g A ^ M (a)) : a G £a,m(R)I 

hmsup < ess ml mj- (81) 

A^O A 

In view of Definition 14.41 and the definition of H~°°(-), the combination of (l80l ) and (|8TI > yields the 
assertion. 

3. a G (— oo, 0]. Here we adapt Renyi's original proof to our case. With the convention 0° := and in 
view of Definition |44]resp. Remark I2T21 it suffices to show that 

/ / Q dA=lim V A 1 '* If fdX] . (82) 

For A > and x G M we define 

5i,a(x) = l supp(M) (x) Yl Vw^aL /QdA 



and 



52,a(x) = lsupp^Cx) l--i (a) (x)A a fdX 



dA 



Applying Lebesgue's differentiation theorem we obtain gx,A - ^ / a and <72,A - ^ / a a - e - as A — >■ 0. Now 
note that since a < 0, the function x h-> x a is convex on (0, oo), so by Jensen's inequality 



' ' fd\) < 1 ( [ f a dX 
9a» / A V^i'w , 



A 
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for all a G q 1 (K), implying g 2 ,A( x ) ^ 51, A ( x ) for a11 x - Since 92, A > and / g 1A dX = J supp(/t) f a dX G 
(0, oo), we can apply the generalized dominated convergence theorem [28, Chapter 11.4] to obtain 

lim / oi a dX = lim / 02 A dA, 

A^o J ' A^o 7 ' 

which is equivalent to (l82l and, therefore, finishes the proof. □ 



Appendix B 

Proof of Proposition 15.71 

(i) Recall that [a;J denotes the largest integer less than or equal to x G R. In view of Lemma [231 we 
have D^(R) < D®(R). Consequently, we deduce from ||23l Lemma 1] the existence of a constant k > 
(that depends only on r and 5) such that 



D«{R) < D°(R) < DjOogd^J)) < (le R \r r K r (/ kl^W 



r/(r+5) 



Due to R > 1 we obtain 



fl -l7 _ Ve-l 



r 



which yields the assertion with Co = K r ( e^i) • 

(ii) In view of Lemma 1231 it is enough to prove relation ((2TI) for a = — 00. Let I = supp(^), R > 
and qn £ Q with H~°°(qji) < i?- According to Lemma 1231 let us assume w.l.o.g. that all codecells of 
qji with positive \i— mass are intervals. By subdivision of codecells with \i— mass greater than or equal 
to 2e~ R we can assume w.l.o.g. that 

e- R < n(qa\a)) < 2 e - R (83) 

for every a G <7ij(R) with ^(^ 1 (a)) > 0, where the first inequality holds since H~ oc (qji) < R. 
Moreover, for every such a we obtain 

diam(g^)n/)< M |^ a)) (84) 

and we can assume w.l.o.g. that a G H / if q] i 1 (a)) H I ^ (otherwise the distortion can be 

decreased by redefining a). Then we have 



D^qn) = y~] / |x - o| r /(x)(iA(a;) 
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< E / (di a m(q~ 1 (a)nI)) r f(x)dX(x). 

.- . n™\ J a™ 1 Co) 



aeg R (M) J 1r^ 

In view of (l83l and (l84l ) we get 



d Mr) < E L { 



aegjj(K) 
1 



<(/) 



/(s)dA(z) 



L E (/^(^ 1 («))) r+1 



which yields (|2TI ) by taking the infimum over all E Q with H~° c (qji) < R. □ 
Proof of Lemma [Ol Let e > 0. Choose c,t £ (0, oo), such that 

1 - /i(A^t) < | (85) 
with A ct = {x : /(x) > c} n {x : /(x) < i} n [-*, i]. Let 



(1 +r)2 r 

and use Corollary l5.2l to choose Rq > such that 



2 

Now let R>R ,5 = D*{R) > 0, and choose g G Q with < R and |L> M (<?) - < 6. We 

have 



^(9) = E / k-a| r /(x)dA(x 



,-i(a) 

> E c / |x-a| r dA(x). (86) 

Let B(x, I) = [x — I, x + I] for any / > and For every a £ <z(M) define 

s = A(g- 1 (o)rii4c >t )/2. (87) 
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Since A C:t is bounded, we have s a G [0, oo). Moreover, it is easy to show that 

I \x - a\ r d\(x) > I \x-a\ r d\{x) (88) 

Jq- 1 (a)nA c ,t J B(a,s a ) 

(see, e.g., (£l Lemma 2.8]). Using (l87l) we compute 

/ \x- a\ r d\(x) = p£ = -Jf-\( q -\a) n A c , t ). 

JB{a,s a ) 1 + r 1 + r 



Combining ([88j> and ([89]> with ([861) we obtain 

c 



r 



D,{q) > — Y, A(?- 1 (a)n^, t )a a 



7J^ E A(< ? - 1 (a)n^, i )(2 Sa )' 



> E ACg-HaJn^t) 1 ^ 



Using (1871 ) we get 



a£4 

> k- sup A(<? _1 (a) n A Cit ) 1+r . 

On the other hand the choice of S and the monotonicity of D^(-) yields 

D^q) < 2D«(R) < 2D°(R ) 

Thus we deduce 

k- sup \(q- 1 (a)nA Cit ) 1+r <2D%(R 



Also, since / is upper bounded by t on A c>t , 

max fi(q~ 1 (a) n A c t) < i- sup A(g _1 (a)fl4 Ci i) 



i 



(89) 



£ ( HH) <5- <90) 

With dSD and (O we finally obtain 

max ^(g _1 (a)) < max /x(g _1 (a) n Aw) + 1 - n{A c A < e, 

aeg(M) aGg(M) 

which proves (J22]). Now, additionally, let i(f) > 0. Let a € g(R) with ^(c/ _1 (a)) > 0. By Lemma 1231 
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we can assume, that q~ l {a) is an interval. Thus we obtain 

A(g -1 (o) n Ii) = diam(g _1 (a) n U 
for every i G {1, . . . , m}. Together with Ii C supp(/z) we deduce 



H(q-\a))>n(q-\a)nii) = / f(x)dX(x) 

= i(f)^am(q-\a)nli) (91) 

for every i G {1, . . . , m}. Relation (l23l follows now immediately from (l22l ) and d9T| ). □ 

Proof of Lemma \5A\ Let a G [— oo, 0]. Then by Lemma [231 for any 7 > 1 there exists a quantizer q 
with H®(q) < R such that each cell of q is an interval with positive fi— mass (and thus q(M) = N q ) and 

L> M (<?)< 7 (92) 
According to definition (O we obtain in case of a > —00 that 

^ /i(g _1 (a)) Q < e ( 1 - a ) R . (93) 

aeg(M) 

We deduce ^ ^ ^ 

e R >( £ / i(g- 1 (a))(l// i ((Z- 1 (o))) 1 - a ) ° >n (94) 



where n is the number of codepoints of q and the second inequality follows from Jensen's inequality 
applied to the concave function x h-» x 1 '^ 01 '. In case of a = —00 we obviously have n < 00. We get 



1= ^2 M(5 X (a)) > n • min{/i(g 1 (a)) : a G > ne" 

aeg(M) 



Hence, n < e R for every a G [— 00, 0]. Let supp(/z) = [c, d], where —00 < c < d < 00. For every 
a G g(M.) let m a denote the midpoint of q^ 1 (a) Pi [c, d]. As in the proof of Lemma 1531 we obtain 



D M = Yl I \x-a\ r f{x)d\{x) 

^2 i(f) / \ x - m a\ T d\(x) 



> 



lq '(a)'1<V 

2 i(/)(2 r (l +r))- 1 diam( (? - 1 (G) fl [c,d]) r+1 . 
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Clearly (cf. 



M(9 " 1(a)) <diam(g- l (a)n[c,d|)<' l(9 " 1(a)) 



«(/) w v-y^ jy- 

for every a € <7(R) with fi(q^ 1 (a)) > 0. Thus we deduce from the convexity of x i-> x r+1 that 



D M (g) > i(/)(2^(l + r))- 1 S (/)- r - 1 £ (^r 1 («))) r 



+1 



n 



> i(/)(2 f '(l + r))- 1 S (/)-'- 1 E(l/nr +1 

i=l 

= i(/)(2 r (l + r))- 1 S (/)- r - 1 n- r . 
Combining (|92l > and Proposition 15.11 (h) we obtain 

i{fW{l + r))- 1 ^)-'- 1 ^ < 7^e~^. (95) 
Because 7 G (1, 00) was arbitrary, inequality d95l > remains valid if we set 7 = 1. Hence we obtain 



*(/) \ r / 1 \ „fl 



which yields (I25I ). 

Now assume a G (—00, 0). We will modify g such that the new quantizer is in K,r and it still satisfies 
the rate constraint, while its distortion does not exceed that of q. Let 

p = max{^(g _1 (a)) : a G q(M.)} > 

and a p G g(R) such that ^(q^ 1 (a p )) = p. If H^(q) < R we can subdivide the cell q ,_1 (a p ) into two 
cells with equal /i— mass, such that the entropy increases by 



^logfW £ MTV))") 

+ T ^io g (2( P /2) a + £ M^wr) >0. 



aeg(R)\{a p } 

If we take the optimal quantization points for the two new cells, the new quantizer does not increase the 
quantization error. As long as the entropy is lower than R we repeat this procedure. Hence there exists a 
modified quantizer (also denoted by q) satisfying 

e (l-a)R _ e (l-a)H2(q) < ^l-a _ (%) 
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Note that d93l ) and d94l ) remain valid also for this modified quantizer. Consequently, 



< Ce R < card(g) < e R < oo. (97) 

Thus we deduce 

e (l-a)R > e (l-a)H«(«) = n{q- l {a)) a > card(g) • p a > Ce R p a , 



which implies 

Together with (|96]> and R > log( 2l ~ ( ^~ 1 ) we obtain 

e (l-a)R e (l-a)R 

1 < 



< 



e (l-a)H«(q) e (l-a)R _ f e {l-a)R _ e (l-a)H«(q)\ 
e (l-a)R 



e (l-a)R _ Ql-a - 

e (l-a)fl 



- e (l-o)fl _ ( 2 l~a _ i)(7-l e -fl e (l-a)fl ' (98) 

In view of d98l ) and d97l > we conclude that g E /Cr, which proves (l26l ). □ 

Proof of Lemma 1331 Recall the definition (l24l) of constant C. Fix k E (0,(7). Let Rq > such that 
(7e R — (m — 1) > Ke R for every i? > Rq. According Lemma [531 in the definition of D^(R) it suffices 
w.l.o.g. to consider for R > Ro only those quantizers q E Hr satisfying 

sup{diam(g~ 1 (a) n I) : a E g(R)} < diam(J)/2m. (99) 



In view of Lemma 15.41 it suffices to show that for R > Rq any quantizer q E T~Lr that satisfies 
can be modified such that the distortion of the new quantizer q does not exceed that of q and it satisfies 
q E JCr(k) and 



2mf{diam(<T i (a) n I) : a E 5(g)} > inf{diam(g~ I (a)) : a E 

According to the upper bound d99l we always have A(g) 7^ 0. If S(q) = 0, then the assertion is 
obvious. Hence, let 5(g) ^ 0. Let us assume w.l.o.g. that //(g _1 (6)) > and that (see Lemma |23T ) 
g _1 (&) is an interval for every b E g(M). For every a E S(q) let 7^ iV(a) C g(K) \ {a} be the set 
of neighbor points, i.e., for every b E N(a) we have either supg _1 (6) = inf g -1 (a) or inf g _1 (6) = 
supg _1 (a). Due to d99) we know that N(a) n 5(g) = 0. Moreover, 7Y(a) C A(q) and card(iV(a)) = 
2. Fix i a E {1, ... ,m — 1} such that q^ 1 (a) C Jj a U ij a +i. Because a E 5(g), we have Ai = 
diam(g _1 (a) n IjJ > and A 2 = diam(g _1 (a) n Ii a +i) > 0. Moreover, diam(g _1 (a)) = Ai + A 2 . 
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Let 61 € N(a) such that inf(g -1 (a)) = sup(g _1 (6i)) and let b 2 G N(a) such that inf(g _1 (6 2 )) = 
sup((7 _1 (a)). Next we will show that 

infCg-^o)) + -y < a < infCg-^o)) + Ai + y. (100) 

To see this, one recognizes that a has to be optimal for /i(-|g _1 (a)). As a consequence (see, e.g., [0 
Lemma 2.6 (a)]), a G [inf (q~ 1 (a)), sup(g -1 (o))]. Moreover, a has to be a stationary point (see (9l 
Lemma 2.5]), which yields 

f \x - a\ r ~ l dn{x) = I Ix-a^dnix). (101) 

■/[inf^" 1 [a)),a] J [a,sup(q- 1 (a))] 

Now let us assume that the first inequality in dlOOl) does not hold. Hence, 

a<inf(g~ 1 (a)) + Ai/2. (102) 

Note that sup(J ia ) = inf q~~ l {a) + Ai and that sup(ijj + A 2 = sup(g -1 (a)). From dlOll ) and A 2 > 
we get 

/ |x - ar 1 d/i(x) > / \x-a\ r - l dfi(x). (103) 

■/[inf^" 1 (a)), a] J [a,inf g" 1 (a)+Ai] 

Because the density of is constant on [inf((? _1 (a)), inf (q~ 1 (a)) + Ai] we obtain from (11031) that 
a > inf(g _1 (a)) + Ai/2, which contradicts (11021) . Thus we have proved the left inequality in (1 100b . 
Similarly, we deduce from Ax > and dlOll ) the right inequality in dlOOl ). 

Recall that \i has constant density on g -1 (&i); i = 1, 2. Again by stationarity dlOll ) we obtain 

61 = inf g -1 (a) - diam(g _1 (fci))/2 (104) 

and 

62 = inf qf-^o) + Ai + A 2 + d[&m(q- 1 (b 2 ))/2. 

LetA = Ai+A 2 . Next we show that w.l.o.g. we can assume 2 A > min(diam(c7 _1 (6i)), diam(g _1 (62)))- 
Assume to the contrary that 

2A < min(diam(c7" 1 (6i)),diam(c7" 1 (6 2 ))). (105) 
Then we have diam((? -1 (6i)) > 2A > 2Ai + A 2 , and applying (1 1041) we get 

inf q- 1 {a) - h > inf q~ l (a) + A 1 + -y - inf g-^a) 
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Hence, (1 1001) implies 

infg -1 (a) - 61 > a-infg _1 (a). (106) 

Similarly we obtain 

62 — supg _1 (a) > supg _1 (a) — a. (107) 
In view of d 105b and by the definition of \x we have 

diam(/) Kq'Hh)) = diamCrt&i)) • s iQ > 2A • s ia 



m 



and 



Moreover, 



Thus we obtain 



(1)<UJ1(/ ^(9~ 1 (^)) = diam(g- 1 (& 2 )) • s io +i > 2A ■ s la+1 . 



m 



diam(I) 

( a ) = A i s i a + A 2Si a +i < 2A max s la , s io+ i}. 

m 



m(<T») < va^{q-'{bx))^{q- l {b 2 ))) (108) 



as long as (11051 ) holds. Thus, in view of (11061) and (1 107b . we can modify q by increasing the codecell 
q~ 1 (a), which yields a reduction of the quantization error and a non-increasing entropy of q (due to 
a < 0, as long as d 108b holds, the entropy is a non-decreasing function of the left endpoint of the cell 
q^ 1 (a) and a non-increasing function of the right endpoint of q~ l (a)). The codecell can be expanded this 
way until 2A = min(diam((7 _1 (&i)), diam(g -1 (&2))) holds. Note that independent of this modification 
q remains an element of T~Lr. Thus we can assume w.l.o.g. that 

2A > min(diam(^ 1 (6i)),diam(g" 1 (6 2 ))). (109) 

If q E JCr(k), then the proof is finished. Hence, let us assume that q £ Kr(k). We will show that q can 
always be modified such that the new quantizer belongs to JCr(k) and still satisfies relation ( 11091 ). We 
proceed as in the proof of relation d26l) . Let 



W(q) = {a G q(R) : ^(a)) = max{^q- l (b)) : b G A(q)}}. 

We subdivide one by one the cells g _1 (a) with a £ W(q) and p = fi(q~ 1 (a)) as in the proof of (1261) 
in Lemma 15.41 Note, that the entropy of the quantizer will exceed any given bound if we repeat the 
subdivision process enough times. We stop this process with a quantizer q that satisfies relation d96l ). 
Now recall that Ce R — (m — 1) > ne R if R > Ro by the definition at the beginning of the proof. Thus, 
withp = fi(q~ 1 (a)), we have 

& (l-a)R > e (l-«)H-(g) > ( car d(g(R)) - ( m - \))p a 

> (Ce R - (m - 1)K > Ke^p". 
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Now the inequality e ( 1_a ^ > me R p a allows us to perform steps identical to the ones in the chain of 
inequalities (l98l) and we obtain that the quantizer belongs to JCr(k). Obviously, (11091 ) is still in force for 
q and the proof is complete. □ 



Appendix C 

Proof of Proposition \6. 21 For every i £ {1, . . . , m} choose a quantizer qi € Q for /ij with H^. (%) < Ri. 
Let 

Ji = {ae 9i (R) : //(CV) n Ai) > 0}. 

Let Ii C N be an index set of the same cardinality as J, and for every k G Ii choose a^k G (M) such 
that Jj = {a^fc : A; G Ij}. Let 

iV = M\U™ 1 U fceJt gr 1 (o i)fc )nA. 
Note that /z(iV) = 0. Now we define the quantizer q by the codecells 

{iV} U {q^ 1 (a i ,k) DA i :i = l,...,m;keI i } 



and corresponding codepoints 

{0} U {a i<k : i = 1, . . . ,m;k e Ii}. 



Note that despite our general assumption, the codepoints now are not necessarily distinct. Recall the 
convention 0° = 0. Since /u(iV) = 0, the definition of H^{q) yields 

° V=i keu J 

\*=1 aegi(K) / 



s a e (l- a )H^( qi ) 



Since H^.(qi) < Ri, we obtain in both cases (a < 1 and a > 1) that 
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Now it is easy to check that H°(q) < R is satisfied if either (|28T ) or d29l ) holds. Further we deduce 

< = / \x - q(x)\ r d^x) 

m „ m 

= E Si / \x - qj{x)\ r dfijjx) = y^SjD^jqi). 
i=i ^ i=i 

Taking the infimum on the right hand side of above inequality yields the assertion. □ 

Proof of Lemma lOl From Definition 16. II we have s € (0, 1). Let R > and 5 > 0. Let q £ Q with 
< ^ and 5 + > D^(q). We obtain 

5 + D«(R) > D^q) >sj\x- q(x)\ r dfi io (x). (110) 
Since a £ [0, 1), we deduce 



i? > H*{q) = T ±-\og[ (E^(9 _1 (o)) 

\a6<?(R) \i=l 

^ T^ log ( E (WtV)))*) 



" Iog(*) + fl« (g). 



1 - a ~~ ov ~ / ' ~ Wo 
Because (5 was arbitrary we get from (1 1 101) that 



which yields 



e^OR) > seKi^^eK^-i^^))^ f R - — log( S 



1 — a 



o 



^0 V 1 - a 

and therefore proves (I3TT) . 

Now let a € [0, r + 1) \ {1} and fix R > 0, such that 

Rq > max{— log(tj) : i = 1, . . . , m}. 
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For any R > Rq let Ri = R + log(ij) > 0, i = 1, . . . , m. We obtain 



m 

^sfei 1 -^ = e (i-«)-R (in) 



ifaG[0,r + l)\{l}. Indeed, (| 1 1 1 b is equivalent to X^I= l s f *i Q = 1. But this equation is satisfied by 
the definition of ij. Applying Proposition 16 .2 1 we obtain 



m 

i=l;i7^io 



Thus we can compute 



m 

e^Cfl) < e^ Sj D" o ( J R io )+ £ e rR Sl D^(R % ) 

m 

= e r ( R - Ri o) se rR *oD%. o (R io )+ e r{R - R ^ Si e rR > D^Ri) 

m 

= str;e rR *o D « io (R to )+ £ s^e^D^R,). (112) 

i=l;i^io 



Because all terms in (11121) are nonnegative we obtain (l32l) . □ 



Appendix D 

Lemma D.l. Let m G N an<i X^i=i s « = 1 s « > 0/or every i € {1, . . . , to}. Le? the probability 
measure p, be supported on a bounded interval I such that n = X^I=i s iU(Ii) where the I{ are intervals 
of equal length A(I)/m that partition I. Let a € (— oo,0) and (R n ) n ^fq be an increasing sequence of 
positive numbers such that R n — > oo as n — > oo. Then for every sequence (g n )neN of quantizers with 
Qn £ 0R n > relation f l?8l ) holds. 

Proof. Recall from (T27T ) the definition of A{q) and S{q). For any n € N 

p (l-a)H«(q n ) 

1 < 



^{ln l ( a )) a + EaeS(g n )^(^n 1 ( a )) a 
'EaeA(q n )V(<ln 1 (a)) a 

c&id(S(q n )) ■ supiniq' 1 (a)) a : a € S(q n )} 



< 1 + 



< l + (m-l) 



SUPM^n^ 13 ))" : Q € g(gn)} 

K ( ln 1 {a)) a 
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[+(>i _ 1)M K i .-M):^ %■,)})'. (U3) 

M(g n (a)) Q 



Now let 



and 



^ = min< > A(/)/m : * G ^' ' ' ' ' ^ ' > ° 



h 2 = max<J ^ — : i G {1, . . . , m\ \ > 0. 
A(l)/m 



Since g n € we have 

1 < 1 I (m l)(h /2) a ^^^^Ml : a £ A (ln)}) a 

< 1 + (m - l)(^/2fe 2 )" ( m Wdiam( g - 1 (a)) = a € A( gw )»" (n4) 

E aG yl( g „) dlam (9» ( a ))" 

Fix i = i(n) G {1, . . . , m} and b G -A(g n ) n such that 

diam( g ~ 1 (6)) = min{diam((;~ 1 (a)) : a G vl(g n )}. (115) 

From Proposition 17.21 and by [9, Example 5.5] we know that all codecells (?^ 1 (a) with a G A(q n ) n J, 
can be assumed to have equal length. Because q n G Gr„ C /Cj? n we obtain lim n _ s . 0O H®(q n ) = oo. In 
view of (II 151 ) we thus get card(^4((7 n ) n Ij) — >• oo as n — > oo. From (11131 ) and (11141 ) we deduce 

^ ^ e (i-a)H«( 9n ) ^ _ ^ ( m - l)(/ ll /2/i 2 )«(diam(g- 1 (6))) Q 

(m - l)(h 1 /2h 2 ) a 

Again from <? n G C K,r u we have lim n ^oo e (l-a)CR»--H£ = 1, which yields together with 
(II 161 ) the assertion. □ 

Let m > 2 and si, . . . , s m G (0, l) m with s « = 1- F° r (vi,..., v m ) G (0, oo) m and a G 

(— oo, oo) \ {1} we define 

m 

f(vi, ...,v m ) = s i v i r 

i=l 

and set t { = s] /a2 (j2]Li s?) » = 1, • • • , m as in ©■ 
Lemma D.2. Tfo G (— oo, 1), 

m 

F(ti,...,t ro )=inf{F(«i,...,« m ) : (vi,...,« m ) G (0,oo) m ; £ sf v\~ a = 1}. 

8=1 
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Proof. Let Xi = sf v, a . We calculate 



Vi = (x i s i 

and 

m 

F(vi,...,v m ) = ^2si(xiS^ a )^ 
i=i 

l-Q+ar 



^ ] a i 1 x i — '■ G{x\, . . . , x m ). 



i=l 



Applying [9, Lemma 6.8] we deduce that G attains its minimum on (0, oo) m subject to the constraint 
Y1T=1 x i = 1 at the P oint (VU ■ ■ ■ , Urn) With 



s i_ 

l-a+ar 



s 1_t * , 

,, : / = !» 



for every i G {1, . . . , m}. Hence, F attains its minimum subject to the constraint YaLi s ? v l ° = 1 at 

_ i 

the point (w\ , . . . , w m ) with = (yiS i a ) !-« for every i G {!,..., m}. We deduce 



1-a + r —Q 



which yields Wi = U. □ 
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